feat: keep embeddings fresh after builds and purge orphaned vectors - #599
feat: keep embeddings fresh after builds and purge orphaned vectors#599SHudici wants to merge 1 commit into
Conversation
Embeddings were write-once: nothing refreshed them after a build, so semantic search silently decayed as code changed, and vectors for deleted or renamed nodes kept surfacing as ghosts. Two additions: - EmbeddingStore.purge_orphans() deletes vectors whose qualified_name no longer exists in the nodes table (both tables share one SQLite file). embed_all_nodes() now calls it, so a manual embed also cleans up ghosts. - refresh_embeddings(graph_store) runs after every build-time post-process: the shared run_post_processing() pipeline (watch mode, eval runner) gains it as a fifth step, and the tool pipeline used by the CLI build command and the MCP build_or_update_graph / run_postprocess tools invokes it at the "full" postprocess level. run_postprocess grows an embeddings flag (default True) alongside its existing flows/communities/fts knobs. The refresh is opt-in by construction: it only acts when the graph already contains embeddings (the user ran embed at least once), and only with the exact provider identity that produced them (stored in the per-row provider tag). If the tagged provider cannot be resolved (missing extras or env vars) or resolves to a different model/endpoint, the refresh is skipped, so a build can never trigger a surprise full re-embed or cloud spend. The hash-incremental embed_nodes() keeps the refresh cheap: only new or changed nodes are re-encoded. Every wiring point downgrades provider or transport errors to a build warning, matching the existing step-isolation contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Maintainer disposition: the orphan cleanup and incremental refresh mechanics are useful, but this branch will not be merged directly. I am preparing a current-main replacement with Stefan Hudici attribution.\n\nOne safety change is required before merge: this patch enables embedding refresh by default in full builds, run_postprocess, and watch-mode post-processing. Once a graph has ever been embedded, routine local rebuilds can therefore send newly changed source text to Google, MiniMax, OpenAI, or an OpenAI-compatible endpoint and incur API cost without an explicit refresh choice on that build. A prior one-time embed is not sufficient consent for permanent automatic network calls.\n\nThe replacement will:\n- keep orphan purging during an explicit embed;\n- keep hash-incremental refresh and exact provider-identity checks;\n- make automatic refresh explicitly opt-in and default-off for builds/watch mode;\n- preserve fail-soft build behavior;\n- test local, cloud-provider, missing-credential, identity-mismatch, orphan, and no-prior-embedding paths.\n\nI will post the replacement PR, full local/CI evidence, merge commit, and attribution here when complete. |
|
Thanks again, Stefan. Final disposition: we adopted the orphan purge, hash-incremental refresh, exact provider-identity checks, and fail-soft reporting in PR #653, preserving your authorship. This source is not being merged directly because it enabled refresh by default after any prior embed; routine build and watch invocations could therefore send changed source to a cloud provider and incur cost without explicit consent for that invocation. The replacement keeps the useful mechanics but makes refresh provider/model-explicit and default-off. PR #653 is merged into |
2 participants
Problem
Embeddings are write-once: nothing refreshes them after a build, so semantic search silently decays as the code changes, and vectors for deleted or renamed nodes keep surfacing as ghost results.
Fix
EmbeddingStore.purge_orphans()deletes vectors whosequalified_nameno longer exists innodes(both tables share one SQLite file).embed_all_nodes()calls it, so a manualcrg embedalso cleans up ghosts. Guarded by asqlite_mastercheck so it is a no-op on databases without anodestable.refresh_embeddings(graph_store)runs as a post-build step on all three postprocess paths:tools/build.py::_run_postprocess(the MCPbuild_or_update_graphtool — skipped atpostprocess="minimal", like the other enrichment steps),run_postprocess(the MCP re-run tool, with a newembeddings: bool = Trueflag exposed throughrun_postprocess_tool), and the sharedpostprocessing.run_post_processing(CLI watch mode, eval runner).The refresh is opt-in by construction — a build can never trigger a surprise full re-embed or cloud spend:
embedat least once).local:<model>,google:<model>,openai:<model>@<host>, ...). If the tagged provider cannot be resolved (missing extras, missing env vars) or resolves to a different model/endpoint, the refresh logs and skips.embed_nodes()keeps the refresh cheap: only new or changed nodes are re-encoded, and orphans are purged in the same pass.Any provider or transport error is downgraded to a build warning, matching the existing step-isolation contract in
postprocessing.py.Testing
New tests cover: orphan purging (with and without a
nodestable), never-embedded → skip, legacy/unresolvable provider tag → skip, provider identity mismatch → skip, and the embed-new-plus-purge happy path ({"embedded": 1, "purged": 1}), plus postprocessing-step isolation (refresh failure → warning, not a failed build). The build-tool path has its own tests: full postprocess invokes the refresh and surfacesembeddings_refreshed/embeddings_purgedcounts, minimal skips it, aNonereturn adds no keys, a provider error degrades to a build warning, and therun_postprocessembeddingsflag is honored. Full suite: 1412 passed / 0 failed.🤖 Generated with Claude Code